Fuzz the executor against a real Postgres, and record what it found - #122

tests/test_sqlglot_workarounds.py was written by hand, one tripwire per bug somebody happened to trip over. That finds the bugs on the paths we walk. It says nothing about the paths we don't — and the two workarounds that silently outlived their cause (#50) are evidence that noticing is not a strategy.

So tools/fuzz generates random SELECTs, runs each against a real PostgreSQL and against the executor, and reports the divergences minimised to the smallest query that still shows them.

createdb pg_mimic_fuzz && python -m tools.fuzz --count 4000

Nothing here runs in CI. It needs a live server, and a suite that fails on a new random seed is a suite people delete. Its output is what runs in CI: fourteen new tripwires in this diff.

Four things it has to get right

Validity, not randomness, is the hard part. A generator that emits garbage spends its budget on syntax errors. generate.py is typed, and knows the handful of things Postgres rejects at runtime rather than at parse time: division and mod() by zero, mod() and two-argument round() on a double precision, a FULL JOIN on a non-hashable condition. About 1% of what it emits is still invalid and those samples are discarded — which is also what lets the shrinker be reckless. Postgres is the validity filter.

Grading, because Postgres and the executor disagree constantly in ways that are not bugs. avg(int) is a Decimal on one side and a float on the other; a comparison that calls that a failure buries the real findings under it. compare.py ranks COUNT/VALUE (always bugs) above EXACT/TYPE, and compares floats within a relative tolerance, since both engines do the same arithmetic in a different order and are allowed to land a few ULPs apart.

Shrinking, without which the tool is useless. 2817 raw failures on the first run were 212 buckets, each otherwise buried under forty lines of generated arithmetic. The moves are deliberately reckless — hoist any node to its own child, replace any node with a literal, drop any clause — because a variant Postgres refuses is rejected exactly like one that stops reproducing. Minimised queries then de-duplicate themselves, two seeds that found one bug converging on one text.

Knowing whose bug it is. The same executor is reachable either side of tables.py, so every finding reports whether it also fails raw sqlglot: also fails raw is upstream reaching our users, raw gets this right is one of our own rewrites breaking it.

What it found

Fourteen new upstream bugs, all still failing against sqlglot main at d138cae. Six are wrong answers rather than refusals, and three of those cannot be detected from the answer:

• count(DISTINCT x) ignores the DISTINCT entirely
• SELECT DISTINCT a, a answers one column; SELECT DISTINCT x.a, y.a over a join answers 3 rows where Postgres answers 9. Same root cause as A derived table's or CTE's own columns are named by qualify(), not as Postgres names them #113 — qualify() names both _col_0 — but the consequence is lost data, not a wrong label
• an outer join's NULL-padded row survives WHERE x.a = 1

Plus CAST(0.5 AS INT) truncating where Postgres rounds (by the same two rules as round()), LIKE 'a\%b' ignoring the escape, EXISTS (subquery) emitting invalid Python, NULL IN (subquery) answering FALSE rather than NULL, and Decimal * float raising TypeError — that last needs no join, subquery or unusual function, just two numeric columns of different types, so it is the likeliest of these to be the first thing a user hits.

Details and the upstreaming split go on #58. Four findings are ours, not upstream — _rewrite_not_in turns a correct answer into a wrong one — and get their own issue rather than riding along here, since this diff adds no test for them.

Text collation is the one finding with no fix

The executor compares strings by code point and Postgres by its database collation, so 'abc' < 'ABC' disagrees on any locale-derived collation, and every ORDER BY divergence on the first run was that. There is no collation for the executor to consult, so this is a modelling gap: documented in docs/limitations.md, with the README noting createdb --locale=C to see past it. With C collation, 13 ordering buckets became 1 real bug.

Notes for review

• docs/limitations.md is the only user-facing change; everything else is a dev tool and tests.
• The tripwires are xfail(strict=True) like their neighbours, so the suite is green on the pinned floor and goes red the moment upstream fixes one.
• tools/ is not packaged ([tool.hatch.build.targets.wheel] packages = ["pg_mimic"]), so none of this ships in the wheel.